Joint flow control mechanism in a telecommunications network

ABSTRACT

A mechanism enabling plural queues in a downstream telecommunications network element to be treated as a single, joint queue for purposes of connection-level flow control. A pointer in at least one queue descriptor points to a queue descriptor in which is maintained a set of shared, joint counters. Other flow control elements are maintained individually with respect to each queue descriptor. This mechanism enables flow control elements associated with a single transmitter queue to flow control plural connections terminating in plural queues associated with a single receiver processor.

RELATED APPLICATION

This application claims benefit of U.S. Provisional Application Ser. No.60/001,498, filed Jul. 19, 1995.

FIELD OF THE INVENTION

This application relates to communications methods and apparatus in adistributed switching architecture, and in particular to a joint flowcontrol mechanism in a distributed switching architecture.

BACKGROUND OF THE INVENTION

A Flow Controlled Virtual Connection (FCVC) protocol for use in adistributed switching architecture is presently known in the art, and isbriefly discussed below with reference to FIG. 1. This protocol involvescommunication of status (buffer allocation and current state) on a pervirtual connection, such as a virtual channel connection or virtual pathconnection, basis between individual upstream and downstream networkelements to provide a "no cell loss" guarantee. A cell is the unit ofdata to be transmitted. Each cell requires a buffer to store it.

One example of this protocol involves a credit-based flow controlsystem, where a number of connections exist within the same link withthe necessary buffers established and flow control monitored on aper-connection basis. Buffer usage over a known time interval, the linkround-trip time, is determined in order to calculate the per-connectionbandwidth. A trade-off is established between maximum bandwidth andbuffer allocation per connection. Such per-connection feedback andsubsequent flow control at the transmitter avoids data loss from aninability of the downstream element to store data cells sent from theupstream element. The flow control protocol isolates each connection,ensuring lossless cell transmission for that connection.

Connection-level flow control results in a trade-off between updatefrequency and the realized bandwidth for the connection. High updatefrequency has the effect of minimizing situations in which a largenumber of receiver cell buffers are available, though the transmitterincorrectly believes the buffers to be unavailable. Thus it reduces thenumber of buffers that must be set aside for a connection.

Under certain circumstances, it is desirable to model a singletransmitter queue as a Virtual Path Connection (VPC) providing datacells to plural queues associated with a single receiver processor, eachreceiver queue regarded as Virtual Channel Connection (VCC). Prior artprotocols fail, however, to enable connection-level flow controlelements associated with the single transmitter queue to jointly flowcontrol the plural receiver queues. Rather, each connection fromtransmitter queue to receiver queue must be individually flowcontrolled, resulting in a redundant and bandwidth-inefficient controlmechanism.

SUMMARY OF THE INVENTION

The presently claimed invention provides a mechanism for grouping pluralqueue control elements in a downstream receiver into a single, joint setof flow control elements. This is particularly desirable in thesituation where a single transmitter queue is the source of data cellscommunicated to plural queues all associated with the same receiverprocessor. While the prior art requires that each such connection beflow controlled distinctly, thus resulting in the maintenance ofduplicative flow control elements and the consumption of bandwidththrough the requirement of transmitting parallel update and checkinformation for each connection, the presently disclosed mechanismenables such plural downstream queues to be flow controlled as a unit.

Each queue has associated therewith a data set known as a queuedescriptor. Among the elements of a queue descriptor in a receiver arecounters and register which are used in connection-level flow control.The queue descriptor further includes fields useful for indicating thatcertain flow control information is tracked in a queue descriptor ofanother queue and for pointing to that queue descriptor. Thus, pluralqueues are flow controlled through the use of counters in a common queuedescriptor.

The use of such a mechanism results in a reduced frequency of check andupdate cell transfer, thus reducing overhead bandwidth, whilemaintaining a desired characteristic of connection-level flow control,zero cell loss.

In alternative embodiments, the presently disclosed mechanism is alsoused in conjunction with link-level flow control of a virtual link, witha prioritized buffer mechanism providing prioritized access to a sharedbuffer resource, and with a mechanism capable of guaranteeing a minimum,allocated bandwidth to a connection.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and further advantages may be more fully understood byreferring to the following description and accompanying drawings ofwhich:

FIG. 1 is a block diagram of a connection-level flow control apparatusas known in the prior art;

FIG. 2 is a block diagram of a link-level flow control apparatusaccording to the present invention;

FIGS. 3A and 3B are flow diagram representations of counterinitialization and preparation for cell transmission within a flowcontrol method according to the present invention;

FIG. 4 is a flow diagram representation of cell transmission within theflow control method according to the present invention;

FIGS. 5A and 5B are flow diagram representations of update cellpreparation and transmission within the flow control method according tothe present invention;

FIGS. 6A and 6B are flow diagram representations of an alternativeembodiment of the update cell preparation and transmission of FIGS. 5Aand 5B;

FIGS. 7A and 7B are flow diagram representations of update cellreception within the flow control method according to the presentinvention;

FIGS. 8A, 8B and 8C are flow diagram representations of check cellpreparation, transmission and reception within the flow control methodaccording to the present invention;

FIGS. 9A, 9B and 9C are flow diagram representations of an alternativeembodiment of the check cell preparation, transmission and reception ofFIGS. 8A, 8B and 8C;

FIG. 10 illustrates a cell buffer pool according to the presentinvention as viewed from an upstream element;

FIG. 11 is a block diagram of a link-level flow control apparatus in anupstream element providing prioritized access to a shared bufferresource in a downstream element according to the present invention;

FIGS. 12A and 12B are flow diagram representations of counterinitialization and preparation for cell transmission within aprioritized access method according to the present invention;

FIGS. 13A and 13B illustrate alternative embodiments of cell bufferpools according to the present invention as viewed from an upstreamelement;

FIG. 14 is a block diagram of a flow control apparatus in an upstreamelement providing guaranteed minimum bandwidth and prioritized access toa shared buffer resource in a downstream element according to thepresent invention;

FIGS. 15A and 15B are flow diagram representations of counterinitialization and preparation for cell transmission within a guaranteedminimum bandwidth mechanism employing prioritized access according tothe present invention;

FIG. 16 is a block diagram representation of a transmitter, a data link,and a receiver in which the presently disclosed joint flow controlmechanism is implemented; and

FIG. 17 illustrates data structures associated with queues in thereceiver of FIG. 16.

DETAILED DESCRIPTION

In FIG. 1, the resources required for connection-level flow control arepresented. As previously stated, the illustrated configuration of FIG. 1is presently known in the art. However, a brief discussion of aconnection-level flow control arrangement will facilitate an explanationof the presently disclosed link-level flow control method and apparatus.

One link 10 is shown providing an interface between an upstreamtransmitter element 12, also known as an UP subsystem, and a downstreamreceiver element 14, also known as a DP subsystem. Each element 12, 14can act as a switch between other network elements. For instance, theupstream element 12 in FIG. 1 can receive data from a PC (not shown).This data is communicated through the link 10 to the downstream element14, which in turn can forward the data to a device such as a printer(not shown). Alternatively, the illustrated network elements 12, 14 canthemselves be network end-nodes.

The essential function of the presently described arrangement is thetransfer of data cells from the upstream element 12 via a connection 20in the link 10 to the downstream element 14, where the data cells aretemporarily held in cell buffers 28. Cell format is known, and isfurther described in "Quantum Flow Control", Version 1.5.1, dated Jun.27, 1995 and subsequently published in a later version by the FlowControl Consortium. In FIG. 1, the block labelled Cell Buffers 28represents a set of cell buffers dedicated to the respective connection20. Data cells are released from the buffers 28, either throughforwarding to another link beyond the downstream element 14, or throughcell utilization within the downstream element 14. The latter event caninclude the construction of data frames from the individual data cellsif the downstream element 14 is an end-node such as a work station.

Each of the upstream and downstream elements 12, 14 are controlled byrespective processors, labelled UP (Upstream Processor) 16 and DP(Downstream Processor) 18. Associated with each of the processors 16, 18are sets of buffer counters for implementing the connection-level flowcontrol. These buffer counters are each implemented as an increasingcounter/limit register set to facilitate resource usage changes. Thecounters of FIG. 1, described in further detail below, are implementedin a first embodiment in UP internal RAM. The counter names discussedand illustrated for the prior art utilize some of the same counter namesas used with respect to the presently disclosed flow control method andapparatus. This is merely to indicate the presence of a similar functionor element in the prior art with respect to counters, registers, or likeelements now disclosed.

Within the link 10, which in a first embodiment is a copper conductor,multiple virtual connections 20 are provided. In an alternativeembodiment, the link 10 is a logical grouping of plural virtualconnections 20. The number of connections 20 implemented within the link10 depends upon the needs of the respective network elements 12, 14, aswell as the required bandwidth per connection. In FIG. 1, only oneconnection 20 and associated counters are illustrated for simplicity.

First, with respect to the upstream element 12 of FIG. 1, two bufferstate controls are provided, BS₋₋ Counter 22 and BS₋₋ Limit 24. In afirst embodiment, each are implemented as fourteen bitcounters/registers, allowing a connection to have 16,383 buffers. Thisnumber would support, for example, 139 Mbps, 10,000 kilometer round-tripservice. The buffer state counters 22, 24 are employed only if theconnection 20 in question is flow-control enabled. That is, a bit in arespective connection descriptor, or queue descriptor, of the UP 16 isset indicating the connection 20 is flow-control enabled.

BS₋₋ Counter 22 is incremented by the UP 16 each time a data cell istransferred out of the upstream element 12 and through the associatedconnection 20. Periodically, as described below, this counter 22 isadjusted during an update event based upon information received from thedownstream element 14. BS₋₋ Counter 22 thus presents an indication ofthe number of data cells either currently being transmitted in theconnection 20 between the upstream and downstream elements 12, 14, oryet unreleased from buffers 28 in the downstream element 14.

BS₋₋ Limit 24 is set at connection configuration time to reflect thenumber of buffers 28 available within the receiver 14 for thisconnection 20. For instance, if BS₋₋ Counter 22 for this connection 20indicates that twenty data cells have been transmitted and BS₋₋ Limit 24indicates that this connection 20 is limited to twenty receiver buffers28, the UP 16 will inhibit further transmission from the upstreamelement 12 until an indication is received from the downstream element14 that further buffer space 28 is available for that connection 20.

Tx₋₋ Counter 26 is used to count the total number of data cellstransmitted by the UP 16 through this connection 20. In the firstembodiment, this is a twenty-eight bit counter which rolls over atOXFFFFFFF. As described later, Tx₋₋ Counter 16 is used during a checkevent to account for errored cells for this connection 20.

In the downstream element 14, the DP 18 also manages a set of countersfor each connection 20. Buffer₋₋ Limit 30 performs a policing functionin the downstream element 14 to protect against misbehavingtransmitters. Specifically, the buffer₋₋ limit register 30 indicates themaximum number of cell buffers 28 in the receiver 14 which thisconnection 20 can use. In most cases, BS₋₋ Limit 24 is equal to Buffer₋₋Limit 30. At some point, though, it may be necessary to adjust themaximum number of cell buffers 28 for this connection 20 up or down.This function is coordinated by network management software. To avoidthe "dropping" of data cells in transmission, an increase in buffers perconnection is reflected first in Buffer₋₋ Limit 30 prior to BS₋₋ Limit24. Conversely, a reduction in the number of receiver buffers perconnection is reflected first in BS₋₋ Limit 24 and thereafter inBuffer₋₋ Limit 30.

Buffer₋₋ Counter 32 provides an indication of the number of buffers 28in the downstream element 14 which are currently being used for thestorage of data cells. As described subsequently, this value is used inproviding the upstream element 12 with a more accurate picture of bufferavailability in the downstream element 14. Both the Buffer₋₋ Limit 30and Buffer₋₋ Counter 32 are fourteen bits wide in the first embodiment.

N2₋₋ Limit 34 determines the frequency of connection flow-ratecommunication to the upstream transmitter 12. A cell containing suchflow-rate information is sent upstream every time the receiver element14 forwards a number of cells equal to N2₋₋ Limit 34 out of the receiverelement 14. This updating activity is further described subsequently. Inthe first embodiment, N2₋₋ Limit 34 is six bits wide.

The DP 18 uses N2₋₋ Counter 36 to keep track of the number of cellswhich have been forwarded out of the receiver element 14 since the lasttime the N2₋₋ Limit 34 was reached. In the first embodiment, N2₋₋Counter 36 is six bits wide.

In a first embodiment, the DP 18 maintains Fwd₋₋ Counter 38 to maintaina running count of the total number of cells forwarded through thereceiver element 14. This includes buffers released when data cells areutilized for data frame construction in an end-node. When the maximumcount for this counter 38 is reached, the counter rolls over to zero andcontinues. The total number of cells received by the receiver element 14can be derived by adding Buffer₋₋ Counter 32 to Fwd₋₋ Counter 38. Thelatter is employed in correcting the transmitter element 12 for erroredcells during the check event, as described below. Fwd₋₋ Counter 38 istwenty-eight bits wide in the first embodiment.

In a second embodiment, the DP 18 maintains Rx₋₋ Counter 40, a counterwhich is incremented each time the downstream element 14 receives a datacell through the respective connection 20. The value of this counter 40is then usable directly in response to check cells and in the generationof an update cell, both of which will be described further below.Similar to the Fwd₋₋ Counter 38, Rx₋₋ Counter 40 is twenty-eight bitswide in this second embodiment.

There are two events in addition to a steady state condition in theconnection-level flow controlled protocol: update; and check. In steadystate, data cells are transmitted from the transmitter element 12 to thereceiver element 14. In update, buffer occupancy information is returnedupstream by the receiver element 14 to correct counter values in thetransmitter element 12. Check mode is used to check for cells lost orinjected due to transmission errors between the upstream transmitter anddownstream receiver elements 12, 14.

In the accompanying figures, connection level counters are augmentedwith "[i]" to indicate association with one connection [i] of pluralpossible connections.

Prior to any activity, counters in the upstream and downstream elements12, 14 are initialized, as illustrated in FIG. 3A. Initializationincludes zeroing counters, and providing initial values to limitregisters such as Link₋₋ BS₋₋ Limit and Link₋₋ Buffer₋₋ Limit. In FIG.3A, Buffer₋₋ Limit[i] is shown being initialized to (RTT*BW)+N2, whichrepresents the round-trip time times the virtual connection bandwidth,plus accommodation for delays in processing the update cell. As forLink₋₋ N2₋₋ Limit, "X" represents the buffer state update frequency forthe link, and for N2₋₋ Limit[i], "Y" represents the buffer state updatefrequency for each connection.

In steady state operation, the UP 16 of the transmitter element 12determines which virtual connection 20 (VC) has a non-zero cell count(i.e. has a cell ready to transmit), a BS₋₋ Counter value less than theBS₋₋ Limit, and an indication that the VC is next to send (also in FIGS.3A and 3B).

The UP 16 increments BS₋₋ Counter 22 and Tx₋₋ Counter 26 whenever the UP16 transmits a data cell over the respective connection 20, assumingflow control is enabled (FIG. 4). Upon receipt of the data cell, the DP18 checks whether Buffer₋₋ Counter 32 equals or exceeds Buffer₋₋ Limit30, which would be an indication that there are no buffers available forreceipt of the data cell. If Buffer₋₋ Counter>=Buffer₋₋ Limit, the datacell is discarded (FIG. 3B). Otherwise, the DP 18 increments Buffer₋₋Counter 32 and Rx₋₋ Counter 40 and the data cell is deposited in abuffer cell 28, as in FIG. 4. The Tx₋₋ Counter 26 and the Rx₋₋ Counter40 roll over when they reach their maximum.

If flow control is not enabled, none of the presently describedfunctionality is implemented. Connections that do not utilize flowcontrol on the link can coexist with connections using link flowcontrol. The flow control accounting is not employed when cells fromnon-flow controlled connections are transmitted and received. Thisincludes both connection level accounting and link level accounting.Thereby, flow control and non-flow control connections can be activesimultaneously.

When a data cell is forwarded out of the receiver element 14, Buffer₋₋Counter 32 is decremented. Buffer₋₋ Counter 32 should never exceedBuffer₋₋ Limit 30 when the connection-level flow control protocol isenabled, with the exception of when BS₋₋ Limit 24 has been decreased andthe receiver element 14 has yet to forward sufficient cells to bringBuffer₋₋ Counter 32 below Buffer₋₋ Limit 30.

A buffer state update occurs when the receiver element 14 has forwardeda number of data cells equal to N2₋₋ Limit 34 out of the receiverelement 14. In the first embodiment in which the DP 18 maintains Fwd₋₋Counter 38, update involves the transfer of the value of Fwd₋₋ Counter38 from the receiver element 14 back to the transmitter element 12 in anupdate cell, as in FIG. 6A. In the embodiment employing Rx₋₋ Counter 40in the downstream element 14, the value of Rx₋₋ Counter 40 minusBuffer₋₋ Counter 32 is conveyed in the update cell, as in FIG. 5A. Atthe transmitter 12, the update cell is used to update the value in BS₋₋Counter 22, as shown for the two embodiments in FIG. 7A. Since BS₋₋Counter 22 is independent of buffer allocation information, bufferallocation can be changed without impacting the performance of thisaspect of connection-level flow control.

Update cells require an allocated bandwidth to ensure a bounded delay.This delay needs to be accounted for, as a component of round-trip time,to determine the buffer allocation for the respective connection.

The amount of bandwidth allocated to the update cells is a function of acounter, Max₋₋ Update₋₋ Counter (not illustrated) at an associateddownstream transmitter element (not illustrated). This counter forcesthe scheduling of update and check cells, the latter to be discussedsubsequently. There is a corresponding Min₋₋ Update₋₋ Interval counter(not shown) in the downstream transmitter element, which controls thespace between update cells. Normal cell packing is seven records percell, and Min₋₋ Update₋₋ Interval is similarly set to seven. Since theUP 16 can only process one update record per cell time, back-to-back,fully packed update cells received at the UP 16 would cause some recordsto be dropped.

An update event occurs as follows, with regard to FIGS. 1, 5A and 6A.When the downstream element 14 forwards (releases) a cell, Buffer₋₋Counter 32 is decremented and N2₋₋ Counter 36 and Fwd₋₋ Counter 38 areincremented. When the N2₋₋ Counter 36 is equal to N2₋₋ Limit 34, the DP18 prepares an update cell for transmission back to the upstream element12 and N2₋₋ Counter 36 is set to zero. The upstream element 12 receivesa connection indicator from the downstream element 14 forwarded cell toidentify which connection 20 is to be updated. In the first embodiment,the DP 18 causes the Fwd₋₋ Counter 38 value to be inserted into anupdate record payload (FIG. 6A). In the second embodiment, the DP 18causes the Rx₋₋ Counter 40 value minus the Buffer₋₋ Counter 32 value tobe inserted into the update record payload (FIG. 5A). When an updatecell is fully packed with records, or as the minimum bandwidth pacinginterval is reached, the update cell is transmitted to the upstreamelement 12.

Once received upstream, the UP 16 receives the connection indicator fromthe update record to identify the transmitter connection, and extractsthe Fwd₋₋ Counter 38 value or the Rx₋₋ Counter 40 minus Buffer₋₋ Counter32 value from the update record. BS₋₋ Counter 22 is reset to the valueof Tx₋₋ Counter 26 minus the update record value (FIG. 7A). If thisconnection was disabled from transmitting due to BS₋₋ Counter 22 beingequal to or greater than BS₋₋ Limit 24, this condition should now bereversed, and if so the connection should again be enabled fortransmitting.

In summary, the update event provides the transmitting element 12 withan indication of how many cells originally transmitted by it have nowbeen released from buffers within the receiving element 14, and thusprovides the transmitting element 12 with a more accurate indication ofreceiver element 14 buffer 28 availability for that connection 20.

The buffer state check event serves two purposes: 1) it provides amechanism to calculate and compensate for cell loss or cell insertiondue to transmission errors; and 2) it provides a mechanism to start (orrestart) a flow if update cells were lost or if enough data cells werelost that N2₋₋ Limit 34 is never reached.

One timer (not shown) in the UP subsystem 16 serves all connections. Theconnections are enabled or disabled on a per connection basis as towhether to send check cells from the upstream transmitter element 12 tothe downstream receiver element 14. The check process in the transmitterelement 12 involves searching all of the connection descriptors to findone which is check enabled (see FIGS. 8A, 9A). Once a minimum pacinginterval has elapsed (the check interval), the check cell is forwardedto the receiver element 14 and the next check enabled connection isidentified. The spacing between check cells for the same connection is afunction of the number of active flow-controlled connections times themandated spacing between check cells for all connections. Check cellshave priority over update cells.

The check event occurs as follows, with regard to FIGS. 8A through 8Cand 9A through 9C. Each transmit element 12 connection 20 is checkedafter a timed check interval is reached. If the connection isflow-control enabled and the connection is valid, then a check event isscheduled for transmission to the receiver element 14. A buffer statecheck cell is generated using the Tx₋₋ Counter 26 value for thatconnection 20 in the check cell payload, and is transmitted using theconnection indicator from the respective connection descriptor (FIGS. 8Aand 9A).

In the first embodiment, a calculation of errored cells is made at thereceiver element 14 by summing Fwd₋₋ Counter 38 with Buffer₋₋ Counter32, and subtracting this value from the contents of the transmittedcheck cell record, the value of Tx₋₋ Counter 26 (FIG. 9B). The value ofFwd₋₋ Counter 38 is increased by the errored cell count. An updaterecord with the new value for Fwd₋₋ Counter 38 is then generated. Thisupdated Fwd₋₋ Counter 38 value subsequently updates the BS₋₋ Counter 22value in the transmitter element 12.

In the second embodiment, illustrated in FIG. 8B, the same isaccomplished by resetting the Rx₋₋ Counter 40 value equal to the checkcell payload value (Tx₋₋ Counter 26 ). A subsequent update record isestablished using the difference between the values of Rx₋₋ Counter 40and Buffer₋₋ Counter 32.

Thus, the check event enables accounting for cells transmitted by thetransmitter element 12, through the connection 20, but either dropped ornot received by the receiver element 14.

A "no cell loss" guarantee is enabled using buffer state accounting atthe connection level since the transmitter element 12 has an up-to-dateaccount of the number of buffers 28 in the receiver element 14 availablefor receipt of data cells, and has an indication of when data celltransmission should be ceased due to the absence of available buffers 28downstream.

In order to augment the foregoing protocol with a receiver elementbuffer sharing mechanism, link-level flow control, also known aslink-level buffer state accounting, is added to connection-level flowcontrol. It is possible for such link-level flow control to beimplemented without connection-level flow control. However, acombination of the two is preferable since without connection-level flowcontrol there would be no restriction on the number of buffers a singleconnection might consume.

It is desirable to perform buffer state accounting at the link level, inaddition to the connection level, for the following reasons. Link-levelflow control enables cell buffer sharing at a receiver element whilemaintaining the "no cell loss" guarantee afforded by connection-levelflow control. Buffer sharing results in the most efficient use of alimited number of buffers. Rather than provide a number of buffers equalto bandwidth times RTT for each connection, a smaller number of buffersis employable in the receiver element 14 since not all connectionsrequire a full compliment of buffers at any one time.

A further benefit of link-level buffer state accounting is that eachconnection is provided with an accurate representation of downstreambuffer availability without necessitating increased reverse bandwidthfor each connection. A high-frequency link-level update does notsignificantly effect overall per-connection bandwidth.

Link-level flow control is described now with regard to FIG. 2. Likeelements found in FIG. 1 are given the same reference numbers in FIG. 2,with the addition of a prime. Once again, only one virtual connection20' is illustrated in the link 10', though the link 10' would normallyhost multiple virtual connections 20'. Once again, the link 10' is aphysical link in a first embodiment, and a logical grouping of pluralvirtual connections in a second embodiment.

The upstream transmitter element 12' (FSPP subsystem) partially includesa processor labelled From Switch Port Processor (FSPP) 16'. The FSPPprocessor 16' is provided with two buffer state counters, BS₋₋ Counter22' and BS₋₋ Limit 24', and a Tx₋₋ Counter 26' each having the samefunction on a per-connection basis as those described with respect toFIG. 1.

The embodiment of FIG. 2 further includes a set of resources added tothe upstream and downstream elements 12', 14' which enable link-levelbuffer accounting. These resources provide similar functions as thoseutilized on a per-connection basis, yet they operate on the link level.

For instance, Link₋₋ BS₋₋ Counter 50 tracks all cells in flight betweenthe FSPP 16' and elements downstream of the receiver element 14',including cells in transit between the transmitter 12' and the receiver14' and cells stored within receiver 14' buffers 28'. As with the updateevent described above with respect to connection-level bufferaccounting, Link₋₋ BS₋₋ Counter 50 is modified during a link updateevent by subtracting either the Link₋₋ Fwd₋₋ Counter 68 value or thedifference between Link₋₋ Rx₋₋ Counter 70 and Link₋₋ Buffer₋₋ Counter 62from the Link₋₋ TX₋₋ Counter 54 value. In a first embodiment, thelink-level counters are implemented in external RAM associated with theFSPP processor 16'.

Link₋₋ BS₋₋ Limit 52 limits the number of shared downstream cell buffers28' in the receiver element 14' to be shared among all of theflow-control enabled connections 20'. In a first embodiment, Link₋₋ BS₋₋Counter 50 and Link₋₋ BS₋₋ Limit 52 are both twenty bits wide.

Link₋₋ TX₋₋ Counter 54 tracks all cells transmitted onto the link 10'.It is used during the link-level update event to calculate a new valuefor Link₋₋ BS₋₋ Counter 50. Link₋₋ TX₋₋ Counter 54 is twenty-eight bitswide in the first embodiment.

In the downstream element 14', To Switch Port Processor (TSPP) 18' alsomanages a set of counters for each link 10' in the same fashion withrespect to the commonly illustrated counters in FIGS. 1 and 2. The TSPP18' further includes a Link₋₋ Buffer₋₋ Limit 60 which performs afunction in the downstream element 14' similar to Link₋₋ BS₋₋ Limit 52in the upstream element 12' by indicating the maximum number of cellbuffers 28' in the receiver 14' available for use by all connections10'. In most cases, Link₋₋ BS₋₋ Limit 52 is equal to Link₋₋ Buffer₋₋Limit 60. The effect of adjusting the number of buffers 28' available upor down on a link-wide basis is the same as that described above withrespect to adjusting the number of buffers 28 available for a particularconnection 20. Link₋₋ Buffer₋₋ Limit 60 is twenty bits wide in the firstembodiment.

Link₋₋ Buffer₋₋ Counter 62 provides an indication of the number ofbuffers in the downstream element 14' which are currently being used byall connections for the storage of data cells. This value is used in acheck event to correct the Link₋₋ Fwd₋₋ Counter 68 (describedsubsequently). The Link₋₋ Buffer₋₋ Counter 62 is twenty bits wide in thefirst embodiment.

Link₋₋ N2₋₋ Limit 64 and Link₋₋ N2₋₋ Counter 66, each eight bits wide inthe first embodiment, are used to generate link update records, whichare intermixed with connection-level update records. Link₋₋ N2₋₋ Limit64 establishes a threshold number for triggering the generation of alink-level update record (FIGS. 5B and 6B), and Link₋₋ N2₋₋ Counter 66and Link₋₋ Fwd₋₋ Counter 68 are incremented each time a cell is releasedout of a buffer cell in the receiver element 14'. In a first embodiment,N2₋₋ Limit 34' and Link₋₋ N2₋₋ Limit 64 are both static once initiallyconfigured.

However, in a further embodiment of the present invention, each isdynamically adjustable based upon measured bandwidth. For instance, ifforward link bandwidth is relatively high, Link₋₋ N2₋₋ Limit 64 could beadjusted down to cause more frequent link-level update recordtransmission. Any forward bandwidth impact would be considered minimal.Lower forward bandwidth would enable the raising of Link₋₋ N2₋₋ Limit 64since the unknown availability of buffers 28' in the downstream element14' is less critical.

Link₋₋ Fwd₋₋ Counter 68 tracks all cells released from buffer cells 28'in the receiver element 14' that came from the link 10' in question. Itis twenty-eight bits wide in a first embodiment, and is used in theupdate event to recalculate Link₋₋ BS₋₋ Counter 50.

Link₋₋ Rx₋₋ Counter 70 is employed in an alternative embodiment in whichLink₋₋ Fwd₋₋ Counter 68 is not employed. It is also twenty-eight bitswide in an illustrative embodiment and tracks the number of cellsreceived across all connections 20' in the link 10'.

With regard to FIGS. 2 et seq., a receiver element buffer sharing methodis described. Normal data transfer by the FSPP 16' in the upstreamelement 12' to the TSPP 18' in the downstream element 14' is enabledacross all connections 20' in the link 10' as long as the Link₋₋ BS₋₋Counter 50 is less than or equal to Link₋₋ BS₋₋ Limit 52, as in FIG. 3B.This test prevents the FSPP 16' from transmitting more data cells thanit believes are available in the downstream element 14'. The accuracy ofthis belief is maintained through the update and check events, describednext.

A data cell is received at the downstream element 14' if neitherconnection-level or link-level buffer limit are exceeded (FIG. 3B). If alimit is exceeded, the cell is discarded.

The update event at the link level involves the generation of a linkupdate record when the value in Link₋₋ N2₋₋ Counter 66 reaches (equalsor exceeds) the value in Link₋₋ N2₋₋ Limit 64, as shown in FIGS. 5B and6B. In a first embodiment, Link₋₋ N2₋₋ Limit 64 is set to forty.

The link update record, the value taken from Link₋₋ Fwd₋₋ Counter 68 inthe embodiment of FIG. 6B, is mixed with the per-connection updaterecords (the value of Fwd₋₋ Counter 38') in update cells transferred tothe FSPP 16'. In the embodiment of FIG. 5B, the value of Link₋₋ Rx₋₋Counter 70 minus Link₋₋ Buffer₋₋ Counter 62 is mixed with theper-connection update records. When the upstream element 12' receivesthe update cell having the link update record, it sets the Link₋₋ BS₋₋Counter 50 equal to the value of Link₋₋ Tx₋₋ Counter 54 minus the valuein the update record (FIG. 7B). Thus, Link₋₋ BS₋₋ Counter 50 in theupstream element 12' is reset to reflect the number of data cellstransmitted by the upstream element 12', but not yet released in thedownstream element 14'.

The actual implementation of the transfer of an update record, in afirst embodiment, recognizes that for each TSPP subsystem 14', there isan associated FSPP processor (not illustrated), and for each FSPPsubsystem 12', there is also an associated TSPP processor (notillustrated). Thus, when an update record is ready to be transmitted bythe TSPP subsystem 14' back to the upstream FSPP subsystem 12', the TSPP18' conveys the update record to the associated FSPP (not illustrated),which constructs an update cell. The cell is conveyed from theassociated FSPP to the TSPP (not illustrated) associated with theupstream FSPP subsystem 12'. The associated TSPP strips out the updaterecord from the received update cell, and conveys the record to theupstream FSPP subsystem 12'.

The check event at the link level involves the transmission of a checkcell having the Link₋₋ Tx₋₋ Counter 54 value by the FSPP 16' every "W"check cells (FIGS. 8A and 9A). In a first embodiment, W is equal tofour. At the receiver element 14', the TSPP 18' performs the previouslydescribed check functions at the connection-level, as well as increasingthe Link₋₋ Fwd₋₋ Counter 68 value by an amount equal to the check recordcontents, Link₋₋ Tx₋₋ Counter 54, minus the sum of Link₋₋ Buffer₋₋Counter 62 plus Link₋₋ Fwd₋₋ Counter 68 in the embodiment of FIG. 9C. Inthe embodiment of FIG. 8C, Link₋₋ Rx₋₋ Counter 70 is modified to equalthe contents of the check record (Link₋₋ Tx₋₋ Counter 54 ). This is anaccounting for errored cells on a link-wide basis. An update record isthen generated having a value taken from the updated Link₋₋ Fwd₋₋Counter 68 or Link₋₋ Rx₋₋ Counter 70 values (FIGS. 8C and 9C).

It is necessary to perform the check event at the link level in additionto the connection level in order to readjust the Link₋₋ Fwd₋₋ Counter 68value (FIG. 9C) or Link₋₋ Rx₋₋ Counter 70 value (FIG. 8C) quickly in thecase of large transient link failures.

Again with regard to FIG. 2, the following are exemplary initial valuesfor the illustrated counters in an embodiment having 100 connections inone link.

    ______________________________________                                        BS.sub.-- Limit      (24') = 20                                               Buffer.sub.-- Limit  (30') = 20                                               N2.sub.-- Limit      (34') = 3                                                Link.sub.-- BS.sub.-- Limit                                                                        (52) =  1000                                             Link.sub.-- Buffer.sub.-- Limit                                                                    (60) =  1000                                             Link.sub.-- N2 Counter                                                                             (66) =  40                                               ______________________________________                                    

The BS₋₋ Limit value equals the Buffer₋₋ Limit value for both theconnections and the link. Though BS₋₋ Limit 24' and Buffer₋₋ Limit 30'are both equal to twenty, and there are 100 connections in this link,there are only 1000 buffers 28' in the downstream element, as reflectedby Link₋₋ BS₋₋ Limit 52 and Link₋₋ Buffer₋₋ Limit 60. This is because ofthe buffer pool sharing enabled by link-level feedback.

Link-level flow control can be disabled, should the need arise, by notincrementing: Link₋₋ BS₋₋ Counter; Link₋₋ N2₋₋ Counter; and Link₋₋Buffer₋₋ Counter, and by disabling link-level check cell transfer. Noupdates will occur under these conditions.

The presently described invention can be further augmented with adynamic buffer allocation scheme, such as previously described withrespect to N2₋₋ Limit 34 and Link₋₋ N2₋₋ Limit 64. This scheme includesthe ability to dynamically adjust limiting parameters such as BS₋₋ Limit24, Link₋₋ BS₋₋ Limit 52, Buffer₋₋ Limit 30, and Link₋₋ Buffer₋₋ Limit60, in addition to N2₋₋ Limit 34 and Link₋₋ N2₋₋ Limit 64. Suchadjustment is in response to measured characteristics of the individualconnections or the entire link in one embodiment, and is establishedaccording to a determined priority scheme in another embodiment. Dynamicbuffer allocation thus provides the ability to prioritize one or moreconnections or links given a limited buffer resource.

The Link₋₋ N2₋₋ Limit is set according to the desired accuracy of bufferaccounting. On a link-wide basis, as the number of connections withinthe link increases, it may be desirable to decrease Link₋₋ N2₋₋ Limit inlight of an increased number of connections in the link, since accuratebuffer accounting allows greater buffer sharing among many connections.Conversely, if the number of connections within the link decreases,Link₋₋ N2₋₋ Limit may be increased, since the criticality of sharinglimited resources among a relatively small number of connections isdecreased.

In addition to adjusting the limits on a per-link basis, it may also bedesirable to adjust limits on a per-connection basis in order to changethe maximum sustained bandwidth for the connection.

The presently disclosed dynamic allocation schemes are implementedduring link operation, based upon previously prescribed performancegoals.

In a first embodiment of the present invention, incrementing logic forall counters is disposed within the FSPP processor 16'. Related thereto,the counters previously described as being reset to zero and counting upto a limit can be implemented in a further embodiment as starting at thelimit and counting down to zero. The transmitter and receiver processorsinterpret the limits as starting points for the respective counters, anddecrement upon detection of the appropriate event. For instance, ifBuffer₋₋ Counter (or Link₋₋ Buffer₋₋ Counter) is implemented as adecrementing counter, each time a data cell is allocated to a bufferwithin the receiver, the counter would decrement. When a data cell isreleased from the respective buffer, the counter would increment. Inthis manner, the counter reaching zero would serve as an indication thatall available buffers have been allocated. Such implementation is lesseasily employed in a dynamic bandwidth allocation scheme since dynamicadjustment of the limits must be accounted for in the non-zero counts.

A further enhancement of the foregoing zero cell loss, link-level flowcontrol technique includes providing a plurality of shared cell buffers28" in a downstream element 14" wherein the cell buffers 28" are dividedinto N prioritized cell buffer subsets, Priority 0 108a, Priority 1108b, Priority 2 108c, and Priority 3 108d, by N-1 threshold level(s),Threshold(1) 102, Threshold(2) 104, and Threshold(3) 106. Such a cellbuffer pool 28" is illustrated in FIG. 10, in which four prioritieslabelled Priority 0 through Priority 3 are illustrated as being definedby three thresholds labelled Threshold(1) through Threshold(3).

This prioritized buffer pool enables the transmission of high priorityconnections while lower priority connections are "starved" or preventedfrom transmitting cells downstream during periods of link congestion.Cell priorities are identified on a per-connection basis. The policy bywhich the thresholds are established is defined according to a predictedmodel of cell traffic in a first embodiment, or, in an alternativeembodiment, is dynamically adjusted. Such dynamic adjustment may be inresponse to observed cell traffic at an upstream transmitting element,or according to empirical cell traffic data as observed at theprioritized buffer pool in the downstream element. For example, in anembodiment employing dynamic threshold adjustment, it may beadvantageous to lower the number of buffers available to data cellshaving a priority less than Priority 0, or conversely to increase thenumber of buffers above Threshold(3), if a significantly larger quantityof Priority 0 traffic is detected.

The cell buffer pool 28" depicted in FIG. 10 is taken from the vantagepoint of a modified version 12" of the foregoing link-level flow controlupstream element 12', the pool 28" being resident within a correspondingdownstream element 14". This modified upstream element 12", viewed inFIG. 11, has at least one Link₋₋ BS₋₋ Threshold(n) 100, 102, 104established in association with a Link₋₋ BS₋₋ Counter 50" and Link₋₋BS₋₋ Limit 52", as described above, for characterizing a cell bufferpool 28 " in a downstream element 14". These Link₋₋ BS₋₋ Thresholds 102,104, 106 define a number of cell buffers in the pool 28" which areallocatable to cells of a given priority, wherein the priority isidentified by a register 108 associated with the BS₋₋ Counter 22"counter and BS₋₋ Limit 24" register for each connection 20". ThePriorities 108a, 108b, 108c, 108d illustrated in FIG. 11 are identifiedas Priority 0 through Priority 3, Priority 0 being the highest. Whenthere is no congestion, as reflected by Link₋₋ BS₋₋ Counter 50" beingless than Link₋₋ BS₋₋ Threshold(1) 102 in FIGS. 10 and 11,flow-controlled connections of any priority can transmit. As congestionoccurs, as indicated by an increasing value in the Link₋₋ BS₋₋ Counter50", lower priority connections are denied access to downstream buffers,in effect disabling their transmission of cells. In the case of severecongestion, only cells of the highest priority are allowed to transmit.For instance, with respect again to FIG. 10, only cells of Priority 0108a are enabled for transmission from the upstream element 12" to thedownstream element 14" if the link-level Link₋₋ BS₋₋ Threshold(3) 106has been reached downstream. Thus, higher priority connections are lesseffected by the state of the network because they have first access tothe shared downstream buffer pool. Note, however, that connection-levelflow control can still prevent a high-priority connection fromtransmitting, if the path that connection is intended for is severelycongested.

As above, Link₋₋ BS₋₋ Counter 50" is periodically updated based upon avalue contained within a link-level update record transmitted from thedownstream element 14" to the upstream element 12". This periodicupdating is required in order to ensure accurate function of theprioritized buffer access of the present invention. In an embodiment ofthe present invention in which the Threshold levels 102, 104, 106 aremodified dynamically, either as a result of tracking the priorityassociated with cells received at the upstream transmitter element orbased upon observed buffer usage in the downstream receiver element, itis necessary for the FSPP 16" to have an accurate record of the state ofthe cell buffers 28", as afforded by the update function.

The multiple priority levels enable different categories of service, interms of delay bounds, to be offered within a single quality of service.Within each quality of service, highest priority to shared buffers istypically given to connection/network management traffic, as identifiedby the cell header. Second highest priority is given to low bandwidth,small burst connections, and third highest for bursty traffic. Withprioritization allocated as described, congestion within any one of theservice categories will not prevent connection/management traffic fromhaving the lowest cell delay.

Initialization of the upstream element 12" as depicted in FIG. 11 isillustrated in FIG. 12A. Essentially, the same counters and registersare set as viewed in FIG. 3A for an upstream element 12' not enablingprioritized access to a shared buffer resource, with the exception thatLink₋₋ BS₋₋ Threshold 102, 104, 106 values are initialized to arespective buffer value T. As discussed, these threshold buffer valuescan be pre-established and static, or can be adjusted dynamically basedupon empirical buffer usage data.

FIG. 12B represents many of the same tests employed prior to forwardinga cell from the upstream element 12" to the downstream element 14" asshown in FIG. 3B, with the exception that an additional test is addedfor the provision of prioritized access to a shared buffer resource.Specifically, the FSPP 16" uses the priority value 108 associated with acell to be transferred to determine a threshold value 102, 104, 106above which the cell cannot be transferred to the downstream element14". Then, a test is made to determine whether the Link₋₋ BS₋₋ Counter50" value is greater than or equal to the appropriate threshold value102, 104, 106. If so, the data cell is not transmitted. Otherwise, thecell is transmitted and connection-level congestion tests are executed,as previously described.

In alternative embodiments, more or less than four priorities can beimplemented with the appropriate number of thresholds, wherein thefewest number of priorities is two, and the corresponding fewest numberof thresholds is one. For every N priorities, there are N-1 thresholds.

In yet a further embodiment, flow-control is provided solely at the linklevel, and not at the connection level, though it is still necessary foreach connection to provide some form of priority indication akin to thepriority field 108 illustrated in FIG. 11.

The link level flow controlled protocol as previously described can befurther augmented in yet another embodiment to enable a guaranteedminimum cell rate on a per-connection basis with zero cell loss. Thisminimum cell rate is also referred to as guaranteed bandwidth. Theconnection can be flow-controlled below this minimum, allocated rate,but only by the receiver elements associated with this connection.Therefore, the minimum rate of one connection is not affected bycongestion within other connections.

It is a requirement of the presently disclosed mechanism that cellspresent at the upstream element associated with the FSPP 116 beidentified by whether they are to be transmitted from the upstreamelement using allocated bandwidth, or whether they are to be transmittedusing dynamic bandwidth. For instance, the cells may be provided inqueues associated with a list labelled "preferred," indicative of cellsrequiring allocated bandwidth. Similarly, the cells may be provided inqueues associated with a list labelled "dynamic," indicative of cellsrequiring dynamic bandwidth.

In a frame relay setting, the present mechanism is used to monitor andlimit both dynamic and allocated bandwidth. In a setting involvingpurely internet traffic, only the dynamic portions of the mechanism maybe of significance. In a setting involving purely CBR flow, only theallocated portions of the mechanism would be employed. Thus, thepresently disclosed method and apparatus enables the maximized use ofmixed scheduling connections--those requiring all allocated bandwidth tothose requiring all dynamic bandwidth, and connections therebetween.

In the present mechanism, a downstream cell buffer pool 128, akin to thepool 28' of FIG. 2, is logically divided between an allocated portion300 and a dynamic portion 301, whereby cells identified as to receiveallocated bandwidth are buffered within this allocated portion 300, andcells identified as to receive dynamic bandwidth are buffered in thedynamic portion 301. FIG. 13A shows the two portions 300, 301 asdistinct entities; the allocated portion is not a physically distinctblock of memory, but represents a number of individual cell buffers,located anywhere in the pool 128.

In a further embodiment, the presently disclosed mechanism forguaranteeing minimum bandwidth is applicable to a mechanism providingprioritized access to downstream buffers, as previously described inconjunction with FIGS. 10 and 11. With regard to FIG. 13B, a downstreambuffer pool 228 is logically divided among an allocated portion 302 anda dynamic portion 208, the latter logically subdivided by thresholdlevels 202, 204, 206 into prioritized cell buffer subsets 208a-d. Aswith FIG. 13A, the division of the buffer pool 228 is a logical, notphysical, division.

Elements required to implement this guaranteed minimum bandwidthmechanism are illustrated in FIG. 14, where like elements from FIGS. 2and 11 are provided with like reference numbers, added to 100 or 200.Note that no new elements have been added to the downstream element; thepresently described guaranteed minimum bandwidth mechanism istransparent to the downstream element.

New aspects of flow control are found at both the connection and linklevels. With respect first to the connection level additions andmodifications, D₋₋ BS₋₋ Counter 122 highlights resource consumption bytracking the number of cells scheduled using dynamic bandwidthtransmitted downstream to the receiver 114. This counter has essentiallythe same function as BS₋₋ Counter 22' found in FIG. 2, where there wasno differentiation between allocated and dynamically scheduled celltraffic. Similarly, D₋₋ BS₋₋ Limit 124, used to provide a ceiling on thenumber of downstream buffers available to store cells from thetransmitter 112, finds a corresponding function in BS₋₋ Limit 24' ofFIG. 2. As discussed previously with respect to link level flow control,the dynamic bandwidth can be statistically shared; the actual number ofbuffers available for dynamic cell traffic can be over-allocated. Theamount of "D" buffers provided to a connection is equal to the RTT timesthe dynamic bandwidth plus N2. RTT includes delays incurred inprocessing the update cell.

A₋₋ BS₋₋ Counter 222 and A₋₋ BS₋₋ Limit 224 also track and limit,respectively, the number of cells a connection can transmit by comparinga transmitted number with a limit on buffers available. However, thesevalues apply strictly to allocated cells; allocated cells are thoseidentified as requiring allocated bandwidth (the guaranteed minimumbandwidth) for transmission. Limit information is set up at connectioninitialization time and can be raised and lowered as the guaranteedminimum bandwidth is changed. If a connection does not have an allocatedcomponent, the A₋₋ BS₋₋ Limit 224 will be zero. The A₋₋ BS₋₋ Counter 222and A₋₋ BS₋₋ Limit 224 are in addition to the D₋₋ BS₋₋ Counter 122 andD₋₋ BS₋₋ Limit 124 described above. The amount of "A" buffers dedicatedto a connection is equal to the RTT times the allocated bandwidth plusN2. The actual number of buffers dedicated to allocated traffic cannotbe over-allocated. This ensures that congestion on other connectionsdoes not impact the guaranteed minimum bandwidth.

A connection loses, or runs out of, its allocated bandwidth through theassociated upstream switch once it has enqueued a cell but has no more"A" buffers as reflected by A₋₋ BS₋₋ Counter 222 and A₋₋ BS₋₋ Limit 224.If a connection is flow controlled below its allocated rate, it loses aportion of its allocated bandwidth in the switch until the congestioncondition is alleviated. Such may be the case in multipoint-to-point(M2P) switching, where plural sources on the same connection, all havinga minimum guaranteed rate, converge on a single egress point which isless than the sum of the source rates. In an embodiment of the presentlydisclosed mechanism in which the transmitter element is a portion of aswitch having complimentary switch flow control, the condition of nothaving further "A" buffer states inhibits the intra-switch transmissionof further allocated cell traffic for that connection.

The per-connection buffer return policy is to return buffers to theallocated pool first, until the A₋₋ BS₋₋ Counter 222 equals zero. Thenbuffers are returned to the dynamic pool, decreasing D₋₋ BS₋₋ Counter122.

Tx₋₋ Counter 126 and Priority 208 are provided as described above withrespect to connection-level flow control and prioritized access.

On the link level, the following elements are added to enable guaranteedminimum cell rate on a per-connection basis. Link₋₋ A₋₋ BS₋₋ Counter 250is added to the FSPP 116. It tracks all cells identified as requiringallocated bandwidth that are "in-flight" between the FSPP 116 and thedownstream switch fabric, including cells in the TSPP 118 cell buffers128, 228. The counter 250 is decreased by the same amount as the A₋₋BS₋₋ Counter 222 for each connection when a connection level updatefunction occurs (discussed subsequently).

Link₋₋ BS₋₋ Limit 152 reflects the total number of buffers available todynamic cells only, and does not include allocated buffers. Link₋₋ BS₋₋Counter 150, however, reflects a total number of allocated and dynamiccells transmitted. Thus, connections are not able to use their dynamicbandwidth when Link₋₋ BS₋₋ Counter 150 (all cells in-flight, buffered,or in downstream switch fabric) minus Link ₋₋ A₋₋ BS₋₋ Counter 250 (allallocated cells transmitted) is greater than Link₋₋ BS₋₋ Limit 152 (themaximum number of dynamic buffers available). This is necessary toensure that congestion does not impact the allocated bandwidth. The sumof all individual A₋₋ BS₋₋ Limit 224 values, or the total per-connectionallocated cell buffer space 300, 302, is in one embodiment less than theactually dedicated allocated cell buffer space in order to account forthe potential effect of stale (i.e., low frequency) connection-levelupdates.

Update and check events are also implemented in the presently disclosedallocated/dynamic flow control mechanism. The downstream element 114transmits connection level update cells when either a preferred list anda VBR-priority 0 list are empty and an update queue is fully packed, orwhen a "max₋₋ update₋₋ interval" (not illustrated) has been reached.

At the upstream end 112, the update cell is analyzed to identify theappropriate queue, the FSPP 116 adjusts the A₋₋ BS₋₋ Counter 222 and D₋₋BS₋₋ Counter 122 for that queue, returning cell buffers to "A" firstthen "D", as described above, since the FSPP 116 cannot distinguishbetween allocated and dynamic buffers. The number of "A" buffersreturned to individual connections is subtracted from Link₋₋ A₋₋ BS₋₋Counter 250.

Other link level elements used in association with the presentlydisclosed minimum guaranteed bandwidth mechanism, such as Link₋₋ Tx₋₋Counter 154, function as described in the foregoing discussion of linklevel flow control. Also, as previously noted, a further embodiment ofthe presently described mechanism functions with a link level flowcontrol scenario incorporating prioritized access to the downstreambuffer resource 228 through the use of thresholds 202, 204, 206. Thefunction of these elements are as described in the foregoing.

The following is an example of a typical initialization in a flowcontrolled link according to the present disclosure:

Downstream element has 3000 buffers;

Link is short haul, so RTT*bandwidth equals one cell;

100 allocated connections requiring 7 "A" buffers each, consuming 700buffers total;

3000-700=2300 "D" buffers to be shared among 512 connections having zeroallocated bandwidth;

Link_(--BS) ₋₋ Limit=2300.

If D₋₋ BS₋₋ Counter>=D₋₋ BS₋₋ Limit, then the queue is prevented fromindicating that it has a cell ready to transmit. In the embodimentreferred to above in which the upstream element is a switch havingcomposite bandwidth, this occurs by the queue being removed from thedynamic list, preventing the queue from being scheduled for transmitusing dynamic bandwidth.

For allocated cells, a check is made when each cell is enqueued todetermine whether the cell, plus other enqueued cells, plus A₋₋ BS₋₋Counter, is a number greater than A₋₋ BS₋₋ Limit. If not, the cell isenqueued and the queue is placed on the preferred list. Else, theconnection is prevented from transmitting further cells through theupstream element 112 switch fabric.

Initialization of the upstream element 112 as depicted in FIG. 14 isillustrated in FIG. 15A. Essentially, the same counters and registersset in FIG. 3A for an upstream element 12' (when prioritized access to ashared buffer resource is not enabled), and in FIG. 12A for an upstreamelement 12" (when prioritized access is enabled). Exceptions include:Link₋₋ A₋₋ BS₋₋ Counter 250 initialized to zero; connection-levelallocated and dynamic BS₋₋ Counters 122, 222 set to zero; andconnection-level allocated and dynamic BS₋₋ Limits 124, 224 set torespective values of N_(A) and N_(B). Similarly, on the downstream endat the connection level, the allocated and dynamic Buffer₋₋ Limits andBuffer₋₋ counters are set, with the Buffer₋₋ Limits employing abandwidth value for the respective traffic type (i.e., BW_(A) =allocatedcell bandwidth and BW_(D) =dynamic cell bandwidth). Further, each cellto be transmitted is identified as either requiring allocated or dynamicbandwidth as the cell is received from the switch fabric.

FIG. 15B represents many of the same tests employed prior to forwardinga cell from the upstream element 112 to the downstream element 114 asshown in FIGS. 3B and 12B, with the following exceptions.Over-allocation of buffer states per connection is checked for dynamictraffic only and is calculated by subtracting Link₋₋ A₋₋ BS₋₋ Counterfrom Link₋₋ BS₋₋ Counter and comparing the result to Link₋₋ BS₋₋ Limit.Over-allocation on a link-wide basis is calculated from a summation ofLink₋₋ BS₋₋ Counter (which tracks both allocated and dynamic celltraffic) and Link₋₋ A₋₋ BS₋₋ Counter against the Link₋₋ BS₋₋ Limit.Similarly, over-allocation at the downstream element is tested for bothallocated and dynamic traffic at the connection level. As previouslyindicated, the presently disclosed mechanism for providing guaranteedminimum bandwidth can be utilized with or without the prioritized accessmechanism, though aspects of the latter are illustrated in FIG. 15A and15B for completeness.

As discussed, connection-level flow control as known in the art reliesupon discrete control of each individual connection. In particular,between network elements such as a transmitting element and a receivingelement, the control is from transmitter queue to receiver queue. Thus,even in the situation illustrated in FIG. 16 in which a single queueQ_(A) in a transmitter element is the source of data cells for fourqueues Q_(W), Q_(X), Q_(Y), and Q_(Z) associated with a single receiverprocessor, the prior art does not define any mechanism to handle thissituation.

In FIG. 16, the transmitter element 10 is an FSPP element having a FSPP11 associated therewith, and the receiver element 12 is a TSPP elementhaving a TSPP 13 associated therewith. The FSPP 11 and TSPP 13 asemployed in FIG. 16 selectively provide the same programmablecapabilities as described above, such as link-level flow control,prioritized access to a shared, downstream buffer resource, andguaranteed minimum cell rate on a connection level, in addition to aconnection-level flow control mechanism. Whether one or more of theseenhanced capabilities are employed in conjunction with theconnection-level flow control is at the option of the systemconfigurator.

Yet another capability provided by the FSPP and TSPP according to thepresent disclosure is the ability to treat a group of receiver queuesjointly for purposes of connection-level flow control. In FIG. 16,instead of utilizing four parallel connections, the presently disclosedmechanism utilizes one connection 16 in a link 14, terminating in fourseparate queues Q_(W), Q_(X), Q_(Y), and Q_(Z), though the four queuesare treated essentially as a single, joint entity for purposes ofconnection-level flow control. This is needed because some networkelements need to use a flow controlled service but cannot handle thebandwidth of processing update cells when N2 is set to a low value, 10or less (see above for a discussion of the update event inconnection-level flow control). Setting N2 to a large value, such as 30,for a large number of connections requires large amounts of downstreambuffering because of buffer orphaning, where buffers are not in-use butare accounted for up-stream as in-use because of the lower frequency ofupdate events. This mechanism is also useful to terminate VirtualChannel Connections (VCC) within a Virtual Path Connection (VPC), whereflow control is applied to the VPC.

This ability to group receiver queues is a result of manipulations ofthe queue descriptor associated with each of the receiver queues Q_(W),Q_(X), Q_(Y), and Q_(Z). With reference to FIG. 17, queue descriptorsfor the queues in the receiver are illustrated. Specifically, thedescriptors for queues Q_(W), Q_(X), and Q_(Y) are provided on the left,and in general have the same characteristics. One of the first fieldspertinent to the present disclosure is a bit labelled "J." When set,this bit indicates that the associated queue is being treated as part ofa joint connection in a receiver. Instead of maintaining allconnection-level flow control information in each queue descriptor foreach queue in the group, certain flow control elements are maintainedonly in one of the queue descriptors for the group. In the illustratedcase, that one queue is queue Q_(Z).

In each of the descriptors for queues Q_(W), Q_(X), and Q_(Y), a "JointNumber" field provides an offset or pointer to a set of flow controlelements in the descriptor for queue Q_(Z). This pointer field mayprovide another function when the "J" bit is not set. While Buffer₋₋Limit (labelled "Buff₋₋ Limit" in FIG. 17) and N2₋₋ Limit are maintainedlocally within each respective descriptor, Joint₋₋ Buffer₋₋ Counter(labelled "Jt₋₋ Buff₋₋ Cntr" ), Joint₋₋ N2₋₋ Counter (labelled "Jt₋₋N2₋₋ Cntr"), and Joint₋₋ Forward₋₋ Counter (labelled "Jt₋₋ Fwd₋₋ Cntr")are maintained in the descriptor for queue Q_(Z) for all of the queuesin the group. The same counters in the descriptors for queues Q_(W),Q_(X), and Q_(Y) go unused. The joint counters perform the same functionas the individual counters, such as those illustrated in FIG. 2 at theconnection level, but are advanced or decremented as appropriate byactions taken in association with the individual queues. Thus, forexample, Joint₋₋ Buffer₋₋ Counter is updated whenever a buffer cellreceives a data cell or releases a data cell in association with any ofthe group queues. The same applies to Joint₋₋ N2₋₋ Counter and Joint₋₋Forward₋₋ Counter. In an alternate embodiment of the previouslydescribed flow control mechanism, each Forward₋₋ Counter is replacedwith Receive₋₋ Counter. Similarly, in an alternative embodiment of thepresently disclosed mechanism, Joint₋₋ Forward₋₋ Counter is replacedwith Joint₋₋ Receive₋₋ Counter, depending upon which is maintained ineach of the group queues. Only the embodiment including Forward₋₋Counter and Joint₋₋ Forward₋₋ Counter are illustrated.

Not all of the per-queue descriptor elements are superseded by functionsin a common descriptor. Buffer₋₋ Limit (labelled "Buff₋₋ Limit" in FIG.17) is set and referred to on a per-queue basis. Thus, Joint₋₋ Buffer₋₋Counter is compared against the Buffer₋₋ Limit of a respective queue.Optionally, the Buffer₋₋ Limit could be Joint₋₋ Buffer₋₋ Limit, insteadof maintaining individual, common limits. The policy is to set the sameBuffer₋₋ Limit in all the TSPP queues associated with a single Joint₋₋Buffer₋₋ Counter.

An update event is triggered, as previously described, when the Joint₋₋N2₋₋ Counter reaches the queue-level N2₋₋ Limit. The policy is to setall of the N2₋₋ Limits equal to the same value for all the queuesassociated with a single joint flow control connection.

When a check cell is received for a connection, an effort to modify theReceive₋₋ Counter associated with the receiving queue results in amodification of the Joint₋₋ Receive₋₋ Counter. Thus, the level ofindirection provided by the Joint₋₋ Number is applicable to both datacells and check cells.

At the transmitter element 10, only one set of upstream flow controlelements are maintained. At connection set-up time, the joint connectionis set-up as a single, point-to-point connection, as far as the upstreamelements are concerned. Therefore, instead of maintaining four sets ofupstream elements for the embodiment of FIG. 16, the presently disclosedmechanism only requires one set of elements (Tx₋₋ Counter, BS₋₋ Counter,BS₋₋ Limit, all having the functionality as previously described).

Once a joint flow control entity has been established, other TSPP queuesfor additional connections may be added. To do so, each new queue musthave the same N2₋₋ Limit and Buffer₋₋ Limit values. The queues for theadditional connections will reference the common Joint₋₋ N2₋₋ Counterand either Joint₋₋ Forward₋₋ Counter or Joint₋₋ Receive₋₋ Counter.

As previously noted, when J=1, the Joint₋₋ Number field is used as anoffset to the group descriptor. The Joint₋₋ Number for the groupdescriptor is set to itself, as shown in FIG. 17 with regard to thedescriptor for queue Q_(Z). This is also the case in point-to-pointconnections (VCC to VCC rather than the VPC to VCC, as illustrated inFIG. 16), where each Joint₋₋ Number points to its own descriptor.Implementation for each of point-to-point and the presently describedpoint-to-multipoint connections is thus simplified.

Having described preferred embodiments of the invention, it will beapparent to those skilled in the art that other embodimentsincorporating the concepts may be used. These and other examples of theinvention illustrated above are intended by way of example and theactual scope of the invention is to be determined from the followingclaims.

What is claimed is:
 1. A method of providing joint connection-level flowcontrol between a sending node, having at least one source queue, and areceiving node, having a plurality of destination queues, via acommunications path, each queue having an associated queue descriptor,said method comprising:identifying a joint destination queue descriptoramong said plurality of destination queue descriptors by establishing apointer in each of said destination queue descriptors, said pointerproviding an offset to said joint destination queue descriptor;transmitting a data cell identified by said at least one source queue;buffering and enqueuing said data cell in one of said plurality ofdestination queues; modifying said joint destination queue descriptor inresponse to said buffering enqueuing; and controlling the subsequenttransmission of data cells identified by said at least one source queueto one or more of said plurality of destination queues based upon saidjoint queue descriptor.
 2. The method of claim 1 wherein said step ofidentifying a joint destination queue descriptor further comprisessetting an indicator in each of said destination queue descriptors to afirst value.
 3. The method of claim 1 wherein said step of transmittingfurther comprises modifying said source queue descriptor in response tosaid data cell buffering and enqueuing.
 4. The method of claim 1 furthercomprising steps of:forwarding said received data cell out of saidreceiving node; and modifying said joint destination queue descriptor inresponse to said data cell forwarding.
 5. The method of claim 1 furthercomprising the step of:comparing a value in said joint destination queuedescriptor, indicative of a number of data cells buffered and enqueuedin said receiving node, with a limit value.
 6. The method of claim 5wherein the step of controlling further comprises:returning an updatecell from said receiving node to said sending node upon said bufferedand enqueued value in said joint destination queue descriptor exceedingsaid limit value, said update cell for use in updating said at least onesource queue descriptor.
 7. The method of claim 5 wherein said step ofcomparing comprises comparing a value, in said joint destination queuedescriptor, indicative of a number of data cells buffered and enqueuedin said receiving node, with a limit value, stored within one of saidplurality of destination queue descriptors.
 8. The method of claim 1further comprising the step of:transmitting, from said at least onesource queue, to any of said plurality of destination queues, a checkcell for use in adjusting said joint destination queue descriptor. 9.The method of claim 1 wherein said receiving and sending nodes eachcomprise a communications network switch.
 10. The method of claim 9,wherein said communications network switches each comprise anasynchronous transfer mode switch.
 11. An apparatus for enablingmultiple receiving queues to be flow-controlled as a single, jointqueue, comprising:a sending node having at least one sending queueassociated therewith; a receiving node having said multiple receivingqueues associated therewith; a communications link disposed between saidsending and receiving nodes for conveying data cells from said sendqueue to one of said multiple receiving queues; a queue descriptorassociated with each of said sending and receiving queues; a joint queuedescriptor associated with said single, joint queue for use in providingcollective flow control information for said multiple receiving queuesto said sending node; and a queue descriptor offset associated with eachof said receiving queue descriptors and providing an offset pointer tosaid joint queue descriptor.
 12. The apparatus of claim 11 wherein saidjoint queue descriptor comprises a joint buffer counter for counting anumber of buffers in use by said multiple receiving queues comprisingsaid single, joint queue.
 13. The apparatus of claim 12 wherein each ofsaid receiving queue descriptors comprises a buffer limit value forcomparison against said joint buffer counter.
 14. The apparatus of claim11 wherein said joint queue descriptor comprises an update counter forcounting a number of data cells received by said multiple receivingqueues comprising said single, joint queue after a most recent updatemessage was sent to said sending node.
 15. The apparatus of claim 14wherein each of said receiving queue descriptors comprises an updatecounter limit value for comparison against said update counter.
 16. Theapparatus of claim 11 wherein said joint queue descriptor is modifiablein response to receipt, by said single, joint queue, of a check cellfrom said at least one sending queue.
 17. The apparatus of claim 16wherein said joint queue descriptor further comprises a count of datacells forwarded out of said multiple receiving queues comprising saidsingle, joint queue, andwherein said count of data cells forwarded ismodifiable in response to receipt of said check cell.
 18. The apparatusof claim 16 wherein said joint queue descriptor further comprises acount of data cells received in said multiple receiving queuescomprising said single, joint queue, andwherein said count of data cellsreceived is modifiable in response to receipt of said check cell. 19.The apparatus of claim 11 wherein said receiving node and said sendingnode each comprise a communications network switch.
 20. The method ofclaim 19, wherein said communications network switches each comprise anasynchronous transfer mode switch.
 21. A method for providing flowcontrol over a communications medium between a first node and a secondnode, comprising the steps of:transmitting a plurality of data unitsfrom said first node to selected ones of a plurality of queuesassociated with said second node, each of said plurality of queueshaving a respective descriptor associated therewith; receiving saidplurality of data units from said first node in said plurality of queuesassociated with said second node; adjusting an offset segment in saiddescriptor of each of said plurality of queues to reference a commonsecond node queue descriptor as a joint queue descriptor; maintainingcollective flow control information pertaining to said plurality ofqueues; and utilizing said collective flow control information to saidtransmission of said plurality of data units to control said selectedones of said plurality of queues associated with said second node,whereby a flow controlled data transaction between said first node andany of said plurality of queues associated with said second node resultsin a modification within said joint queue descriptor.
 22. The method ofclaim 21, wherein said step of maintaining further comprises:setting anotifying segment in said descriptor associated with each of saidplurality of queues to a first value.
 23. The method of claim 21 whereinsaid first and second nodes each comprise a communications networkswitch.
 24. The method of claim 23, wherein each of said communicationsnetwork switches comprise an asynchronous transfer mode switch.
 25. Themethod of claim 21, wherein said step of maintaining comprisesmaintaining a single set of collective flow control parametersassociated with said plurality of queues in said joint queue descriptor.26. The method of claim 25, wherein said step of maintaining furthercomprises maintaining in said joint queue descriptor a count of dataunits buffered and enqueued in association with said plurality ofqueues.
 27. The method of claim 21, wherein said step of maintainingfurther comprises maintaining a respective second set of flow controlinformation logically associated with each of said plurality of queues,each of said second sets of flow control parameters logically associatedwith said collective flow control information.
 28. An apparatus forenabling joint flow control, comprising:a communications medium; a firstnode for transmitting data units over said communications medium; asecond node for receiving said data units from said first node over saidcommunications medium; plural queues associated with said second nodefor enqueuing said data units; a queue descriptor associated with eachof said plural queues; a logical joint queue associated with said secondnode; a joint queue descriptor associated with said logical joint queue;and an offset field in each of said queue descriptors providing anoffset to said joint queue descriptor, wherein said second node isoperative to provide collective flow control information pertaining tosaid plural queues to said first node by reference to said joint queuedescriptor.
 29. The apparatus of claim 28, further comprising:a jointflow control indicator field in each of said queue descriptors, whereinsaid joint flow control indicator field provides an indication that eachof said plural queues is to be regarded as belonging to said logicaljoint queue.
 30. The apparatus of claim 28 wherein said first and secondnodes each comprise a communications network switch.
 31. The apparatusof claim 30, wherein said communications network switches each comprisean asynchronous transfer mode switch.
 32. The apparatus of claim 28,wherein said first node is operative to flow control said data units inresponse to said flow control information from said second node.